As the market for video decoders matures, a cost and size of the memory used to buffer pictures in the decoding process is becoming an important factor. Conventional techniques used to reduce the decoding memory for MPEG-2 compliant bitstreams are (i) dropping B-frames and (ii) storing reference frames at a reduced resolution. However, the conventional solutions are far from optimal in terms of a quality versus a memory tradeoff for an H.264-compliant decode. Unlike MPEG-2 and other prior standards, the H.264 decode permits (i) multiple reference frames, (ii) mandatory storage of non-reference pictures for reordering delay only (and not for use as references), (iii) very flexible picture patterns (i.e., Groups of Pictures), (iv) hierarchical Groups of Pictures and (v) reference B-frames.